Exploring Repeatability on Mechanical Turk
نویسنده
چکیده
Crowdsourcing is gaining attention from companies looking for a low-cost, scalable labor force as well as from researchers seeking to leverage human intelligence and study human cognition. Much current crowdsourcing research is related to discovering methods for obtaining high-quality, trustworthy output from the crowd. One element of quality crowd-generated work is repeatability; ie, results are more trustworthy if they remain constant even if gathered at different times or dates. In this work, we propose and explore various factors that could impact repeatability in the context of a concrete experiment.
منابع مشابه
Exploring Normalization Techniques for Human Judgments of Machine Translation Adequacy Collected Using Amazon Mechanical Turk
This paper discusses a machine translation evaluation task conducted using Amazon Mechanical Turk. We present a translation adequacy assessment task for untrained Arabicspeaking annotators and discuss several techniques for normalizing the resulting data. We present a novel 2-stage normalization technique shown to have the best performance on this task and further discuss the results of all tec...
متن کاملExploring Iterative and Parallel Human Computation Processes
Mechanical Turk (MTurk) is an increasingly popular web service for paying people small rewards to do human computation tasks. Current uses of MTurk typically post independent parallel tasks. This research explores an alternative iterative paradigm, in which workers build on each other's work. We run a couple of experiments comparing the efficacy of this paradigm in two different problem domains...
متن کاملShedding (a Thousand Points of) Light on Biased Language
This paper considers the linguistic indicators of bias in political text. We used Amazon Mechanical Turk judgments about sentences from American political blogs, asking annotators to indicate whether a sentence showed bias, and if so, in which political direction and through which word tokens. We also asked annotators questions about their own political views. We conducted a preliminary analysi...
متن کاملExploring Temporal Vagueness with Mechanical Turk
This paper proposes schematic changes to the TempEval framework that target the temporal vagueness problem. Specifically, two elements of vagueness are singled out for special treatment: vague time expressions, and explicit/implicit temporal modification of events. As proof of concept, an annotation experiment on explicit/implicit modification is conducted on Amazon’s Mechanical Turk. Results s...
متن کاملTurKit: Human Computation Algorithms on Mechanical Turk Citation
Mechanical Turk provides an on-demand source of human computation. This provides a tremendous opportunity to explore algorithms which incorporate human computation as a function call. However, various systems challenges make this difficult in practice, and most uses of Mechanical Turk post large numbers of independent tasks. TurKit is a toolkit for prototyping and exploring truly algorithmic hu...
متن کامل